DOCUMENT RESUME 



ED 451 232 



TM 032 460 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Bastick, Tony- 

Reliable and Valid Measurement of Individual's Contributions 
to Group Work. 

1999-05-00 

8p.; Paper presented at the Annual Meeting of the Western 
Psychological Association (79th, Irvine, CA, April 29-May 2, 
1999) . For a related paper from the same author, see TM 032 
456 . 

Reports - Research (143) -- Speeches/Meeting Papers (150) 

MFOl/PCOl Plus Postage . 

♦Accountability; College Students; *Cooperative Learning; 
Criteria; *Evaluation Methods; *Factor Structure; *Group 
Activities; Group Membership; Higher Education; Reliability; 
Student Evaluation; *Student Participation; Validity 



ABSTRACT 



A method for measuring the contribution an individual makes 
to group work is described, and its use is supported through a study of 57 
university students aged from 20 to 46 years working in 8 groups of 4 to 10 
members each. The method recognizes that the most valid sources of 
information on the contribution of each individual to the group work are the 
group members. The study relied on peer evaluation of the percentage 
contribution that each member made to the group work. Group members were also 
asked for a rationale for each evaluation given. The reliability of the 
individual's mark was indicated by the variance of the marks the group member 
received from his or her peers. Additional validity was provided by the 
qualitative agreement of the rationales. The group work was assessed 
separately using usual appropriate criteria. An individual's mark was 
calculated as his or her mean percentage of the group's mark, weighted by the 
number of people in the group. By separating the measurement of process from 
the assessment criteria for the group work, the method highlighted the 
cooperative intragroup dynamics. The method also allowed the identification 
of consistent patterns of correlations within group scores that indicate a 
problem common to students with low achievement in group work that is 
independent of the subject assessed. (Contains 2 tables and 18 references.) 
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Cooperative groupwork offers pedagogical advantages of social learning, 
leadership practice, peer guidance and the sharing of knowledge and 
experience. Teachers who utilize this effective teaching technique also like 
to assess the contribution of each group member and use it for the purposes 
of individual assessment. There are various ways of allotting marks to an 
individual based on the group’s work. Each method brings with it its ovm 
problems and none has sufficient evidence of reliability and validity. These 
problems have detracted from the use of this preferred method for high- 
stakes summative assessment. The method of assessing cooperative 
groupwork presented in this paper offers improved reliability and validity 
evidence that increases the attractiveness of this form of assessment for 
summative decision making. 



The simple method presented in this paper has been used successful with 
university students (n=57) aged from 20 to 46 years working in groups (8 
groups) of size from 4 to 10 in Psychology and in Measurement courses. The 
method recognizes that the most valid source of information on the 
contribution of each individual to. the groupwork is the group members 
themselves. Hence, the method used peer evaluation of the percentage 
contribution that each member had made to the group’s work. It also asked 
for a rationale for the evaluations that were given. The reliability of an 
individual’s mark was indicated by the variance of the marks the group 
member received from his/her peers. Further validity was given by the 
qualitative agreement of the rationales. The groupwork was assessed 
separately on usual appropriate criteria. An individual’s mark was calculated 
as his/her mean percentage of the group’s mark, weighted by the number of 
people in the group. 

By separating the measurement of process from the assessment criteria 
of the groupwork, the method highlights the cooperative intra-group 
dynamics. The method has also allowed the identification of consistent 
patterns of correlations within group scores that indicates a problem common 
to low groupwork achievers that is independent of the subject area assessed. 



Background 

Advantages of 
cooperative learning 



Teachers recognize that cooperative learning is an excellent teaching technique. One of the 
reasons for this is that teachers consider co-operative learning to be useful for their students’ 
social and psychological development as well as for their academic learning (Khattri, 1991). 
Research in social learning indicates that students learn thinking strategies from these 
interactions with their peers (Zimmerman, 1990). Students are energised by cooperative 
groupwork and see distinct learning advantages in this paradigm (Orsmond, 1996). Student 
reactions to the cooperative assessment processes are overwhelmingly positive (Griffin, 
1994). Groupwork has been shown to improve student involvement and ownership, and to 
increase student retention and transfer of learning (Petty, P. 1997). 
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Popularity of 
cooperative learning 



Need to improve 
reliability and validity 
of coopative learning 



Problems with 
attempted solutions to 
assessing individual 
contributions to 
groupwork 



Fitting the assessment 
method to the 
instructional purpose 
of collaborative work 



One way in which group learning works is that it; 

forces learners to adjust their thinking to that of others. When students 
have to think about the alternative viewpoints of group members, they have 
to elaborate and defend their own ideas and debate the merits of their opinions 
to others. This promotes a deeper organization and understanding of their 
own knowledge.” (Tombari & Borich, 1999, p.lOO). 

These recognized advantages of groupwork have increased the popularity of this method of 
teaching so that . . “Classrooms at all levels of education are increasingly emphasizing group- 
based or cooperative learning strategies.” (Airasian, 1994, p.301) 

Teachers who use this technique would also like to award each individual in the group a 
grade that is based on the assessment of the group’s work. However, compared to traditional 
assessment methods, there are many problems with the validity and reliability of assessing 
group assignments (Palchikov, 1986; Powers & Medena, 1984; Salend & Sonnenschein, 
1989). These problems contribute to the difficulties of replacing high-stakes objective tests 
with these pedagogically preferred ‘alternative’ assessments. 

Common, yet inadequate solutions, to the group assessment problem are to assume that 
each student has made an equal contribution and award each student the same grade. This 
is not popular with students, as the assumption is rarely justified. Conway (1993) reports 
that students complain that group scores are an inadequate reflection of their individual 
effort. This solution can also act against collaborative work and cause social loafing (Gibbs, 
1993; Rotfeld, 1998). The other extreme is for the teacher to give detailed roles, guidelines 
and checklists. This is done when a major instructional aim is to teach socialization skills 
rather than to focus on performance and understanding in the content area. This is because 
checklist criteria are usually valued social skills such as coaching, fimction-filling, social 
facilitation and interaction. Examples of criteria for checklists are ‘Reports to group’, 
‘Completes assigned tasks on time’, ‘considers viewpoints of others’, ‘encourages other to 
do well’, and ‘willing to share materials with others’ (Airasian, 1994, p.256; Linn, & 
Gronlund, 1995, p.276). When this is the intention, checklist criteria for these social skills 
are given beforehand so that the groupwork will promote socialization by the students 
working towards these prosocial assessment criteria. Hence, it is necessary to decide on a 
clear instructional purpose of using groupwork, for example to foster prosocial behaviours 
or to improve performance and understanding in a content area, and then ensure that 
collaboration works toward, rather than against, the purposes of the assessment (Webb, 
1995). Once these decisions have been made then sensitive professional judgement should 
temper the degree of teacher intervention, because “.. to push too hard for individual pupil 
solutions and contributions destroys many of the benefits of cooperative problem solving.” 
(Airasian, 1994, p.301). 

If promoting socialization is not an instructional aim then students can use their own criteria. 
Students can be asked to peer mark their colleagues and to give their own criteria or 
justification for the mark they have given. The consistency of these reasons can then be 
used for construct validity of the average mark received by each single group member. This 
was the method used in the research reported in this paper. 




Page 2:6 



4 



Methodology 

Subjects, data and Students taking an undergraduate assessment course and students taking a Masters coiu'se 
methodology in psychology were given group work assignments as part of their summative course 
assessments. The students chose their own groups (sizes 2 to 10) for their own practical and 
personal bonding reasons. Only the results for the students who chose groups of size 4 to 10 
are included in this report, because these group sizes are appropriate to the statistics used in 
the analysis. There were eight such groups which included 57 students, males and females, 
whose ages ranged from 20 to 46 years. 

The measurement method used in this research was to separate the assessment of the final 
group product from the assessment of students’ individual contributions to the group product. 
The products were assessed according to content standards set at the begirming of the 
groupwork. The contributions of individual group members was confidentially peer assessed 
at the end of the groupwork. Results of other studies have shown that students think peer 
assessment is an important part of the group grading process (Keaten & Richardson, 1993), 
and attitude surveys have shown that students perceive this separation of assessment as fair 
and effective (Beard, 1989). 



Separation of 
process and 
product 
assessment 



Fail-safe validation for 
administrative 
accountability 



Students were advised, right at the start, to be sure that other group members knew how 
they were contributing to the group’s work. Each member was asked to keep a log of 
everyone’s contributions. Students were told that at the end of the course they would award 
each other marks for their contributions to the group’s work. These logs were to be used to 
supply the confidential justifications for the marks students would award. The purpose of 
this design was to moderate any personality or popularity effects that might have influenced 
students’ peer assessments, as have been reported by Brown and Knight (1994). The logs, 
which were also collected, were to be used as a failsafe source of validation, in conjunction 
with post-assessment interviews, if that should be necessary. The instructions for the 
confidential feedback forms were as follows: 



Peer assessment Private and Confidential - do not show this information to any other group member. 

instructions when it has been completed, fold it and staple it Put it in an envelope with the other 

share certificates from your group and submit the envelope with the group work. 
Please print your name and ID no. 

Please 

Below are the names of the people in your group. Put a star "■ by your name. In the box by each name write 
the percentage you think that person deserves - including yourself Then, for each person, give your reasons 
why you decided that person deserved the percentage you gave - including yourself When you have 
finished dieck that the total is 100% 



Validity of peer 
assessed contributions. 
and expert assessed 
content 



These peer assessments could be expected to have a higher content validity than those of an 
external assessor. This is because the group members could be expected to know the exact 
content of each member’s contribution to the group’s work better than an external assessor, 
such as the teacher. However, the students may not be so expert as an external assessor in 
the criterion assessment of the content standard. Hence, the content standard of the product 
was assessed in the traditional manner against the criteria that had been given at the start of 
the course. 
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Table 1 illustrates how the product assessment was weighted by the number of students in 
the group and combined with their confidential peer assessments to arrive at each individual’s 
mark. 

Table 1 : Weighting product assessment by group size and sharing total marks according to 
the average peer assessment for each member. 



Calculation of 
individual marks from 
the product 
assessment, weighted 
by the group size, and 
shared according to 
the average peer 
assessment of each 
individual’s 
contribution 



Group 2 
Assessment 


Percentages 
given by 
group members 


Average received 


Std. Dev received 


Raw Individual 
mark received 


Individual mark 


Group Number 
Subject area 


Marks available 


395 


Group®^ for assignment 


79 


10 


Maths 


Number In group 


5 


st/id 


disC'id 


name 


st/id 


21 


22 


23 


60 


62 










21 


95-( 






Nicola 


25.0 


25.0 


24.0 


25.0 


24.0 


24.6 


0.55 


97.2 


97 


22 


95-( 


Confidential 
Names and IDs 


na 


21.0 


19.0 


18.0 


19.0 


20.0 


19.4 


1.14 


76.6 


77 


23 


95-( 


elix 


14.0 


15.0 


17.0 


17.0 


15.0 


15.6 


1.34 


61.6 


62 


60 


97-( 


ndre 


24.0 


23.0 


23.0 


22.0 


22.0 


22.8 


0.84 


90.1 


90 


62 


Q7A 




— j J 


nna 


16.0 


18.0 


18.0 


17.0 


19.0 


17.6 


1.14 


69.5 


70 






% total check = 100% 


100 


100 


100 


100 


100 


100 








Corr sd of given 
with received = 




Means of marks given 


20.0 


20.0 


20.0 


20.0 


20.0 


20.0 


1.0 


79.0 


79.2 


0.74 


St.devs of marks given 


4.8 


4;o 


3.2 


•■-3;5 


3.4 


3.7 


0.3 


14.6 


14.3 



Table 1 shows the results from a group of size five and how the five students’ assessments 
from their confidential forms have been processed. For example, column 2 1 has the five 
marks given by student No.21 these are 25.0, 21.0, 14.0, 24.0 and 16.0 and the ‘% total 
check’ is 100 as required. When the marks have been entered for all five columns, in the 
same row order, then each row holds the marks received by each student. So in this example 
the first row is for student 21 and the marks received for that student are respectively 25.0 
(self-assessed), 25.0 (from student 22), 24.0 (from student 23), 25.0 (from student 60) and 
24.0 (from student 62). The average of this row, 24.6%, is the percentage of the total mark 
that the group has allotted to student 21. To find the final mark for this student we find the 
number of marks that have been made available from the assessment of the performance 
and the number of group members. In our example it is 5x79=395. That is the quality of the 
finished work was independently assessed at 79%. The 79 is multiplied by the number of 
members in the group, 5 in this case, and each student gets their share e.g. student 21 gets 
29.4% of 5 X 79 which is 97% as shown in the last column of the table. 



Internal reliability and 
construct validity of the 
individual's mark 



The variation in the marks received by a group member are used as a measure of the reliability 
of that member’s mark. The reasons given for each of these marks is used as an indication 
of the construct validity of the group member’s mark. 



Method reveals a 
discrimination problem 
with cooperative 
learning 



Findings from the analysis of the marks given and received by the group members showed 
a consistent pattern across all eight groups and content areas. The variation in the marks 
that were given by a group member to the other members of the group was positively 
correlated with the total mark that was received by that group member from the rest of the 
group. It must be remembered that the two confidential processes, (i) giving a mark to 
others and (ii) the average of the marks received, are independent processes that are now 
shown to be statistically, correlated. 
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Table 2, lists the findings from all eight groups to illustrate the consistency of this finding. 
These groups are not ‘samples’ and so it is the effect-size of the correlation that is of interest. 
The significances are given only for completeness. 

Table 2: Showing a consistent positive correlation across groups and subjects of ‘marks 
received’ with ‘variation in marks given’ 



Table illustrating the 
consistency of the 
discrimination problem 





Group 5 


Group 6 


Group 7 


Group 8 


Corr 


.7317 


.9061 


.5598 


.3389 


n 


( 4) 


( 5) 


( 10) 


( 13) 


Sig 


P= .268 


P= .034 


P= .092 


P= .257 





Group 1 


Group 2 


Group 3 


Group 4 


Corr 


.7650 


.7418 


. 9690 


1.0000 


n 


( 10) 


( 5) 


( 5) 


( 5) 


Sig 


P= .010 


P= .151 


P= .007 


P= .000 



This correlation means that the less a group member is able to distinguish between the 
value of the contributions of group members then the lower is the mark independently 
awarded to that group member by the other students. 

Conclusion 

Educators have a clear pedagogic preference for co-operative performance assessments 
over traditional high-stakes forms of objective tests. However, problems of reliability and 
validity associated with the assessment of co-operative performances make it difficult to 
support these educationally preferred forms of assessment for high-stake gate-keeping 
decisions. The method of assessing an individual’s contribution to co-operative groupwork 
detailed in this paper may make these ‘alternative’ assessments more widely acceptable for 
such important summative evaluations. The method detailed here offers (a) evidence of 
reliability which is based on the variation of peer assessed marks received, (b) evidence of 
construct validity which is based on the reasons given for group members’ marks, and (c) 
content validity which is based on group members’ insiders’ knowledge of each individual’s 
contribution to the product content. The logs of individual’s contributions, and the availability 
of post-assessment interviews, offers the failsafe rigour necessary to support the use of 
groupwork for high-stakes summative assessment. In addition, the method indicates that 
students’ low discrimination between components of the group’s performance might be a 
fundamental factor preventing higher level attainment in cooperative learning assignments. 
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